Challenges When Using Labels For Trajectories